Skip to content

[procmgr] Config gates for processes.d auto-start - #54732

Open
jose-manuel-almaza wants to merge 26 commits into
jose/procmgr-reload-gatefrom
jose/procmgr-config-gates
Open

[procmgr] Config gates for processes.d auto-start#54732
jose-manuel-almaza wants to merge 26 commits into
jose/procmgr-reload-gatefrom
jose/procmgr-config-gates

Conversation

@jose-manuel-almaza

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds config gates to dd-procmgr so processes.d definitions can use condition_config_any to auto-start only when Agent config says they should.

Implementation mirrors the Windows legacy SCM startup checks in dependent_services_windows.go and Agent config resolution:

  • YAML lookup (case-insensitive keys, flattened dotted keys, permissive parse fallback, merge keys)
  • Environment bindings (DD_*) with Agent precedence (ignore empty values, no trim before ParseBool, legacy process_config.enabled transforms)
  • Fleet policy merge
  • Derived system_probe_config.enabled (USM/NPM/security knobs, sk-tracer and discovery adjustments)
  • Windows: read DD_* overrides from the core Agent SCM Environment registry when not set in the procmgr process env

Wires gate evaluation into ManagedProcess start/reload paths in the manager.

Stack context: PR 2/4 split from #53249. Builds on #54731 (jose/procmgr-spawn-profiles). ENC[...] secret backend resolution lands in PR 3 (jose/procmgr-secret-backend-gates). Process-agent dual-mode integration lands in PR 4 (jose/procmgr-windows-process-agent).

Motivation

Moving subservices (starting with process-agent) to dd-procmgr requires the supervisor to apply the same start/stop rules as the Agent today. Without config gates, a processes.d entry would always spawn when registered, which breaks parity with legacy SCM and fleet policy.

Describe how you validated your changes

  • Rust unit tests in pkg/procmgr/rust/src/config_gate/ (env bindings, YAML load, system-probe derivations, gate evaluation)
  • Go unit tests for Windows config helpers (pkg/config/setup/config_windows_test.go)
  • Windows procmgr Rust build/tests in CI

Additional Notes

  • No ENC[...] / secret backend resolution in this PR (fleet policy ENC[...] values are not resolved here either).
  • No agent startup, fleet installer, or legacy SCM suppression changes in this PR.
  • Schema comment sync in pkg/config/schema/yaml/process_config.yaml for env binding documentation.
  • Small exports in pkg/system-probe/config/ so procmgr gate derivations stay aligned with Go adjust* logic.

@jose-manuel-almaza
jose-manuel-almaza requested review from a team as code owners August 11, 2026 15:51
@jose-manuel-almaza jose-manuel-almaza added the qa/done QA done before merge and regressions are covered by tests label Aug 11, 2026
@dd-octo-sts dd-octo-sts Bot added internal Identify a non-fork PR team/agent-runtimes labels Aug 11, 2026
@jose-manuel-almaza jose-manuel-almaza added changelog/no-changelog No changelog entry needed and removed team/agent-runtimes labels Aug 11, 2026
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Aug 11, 2026
Comment thread pkg/procmgr/rust/src/config_gate/env_bindings.rs Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this just read the config schema instead of duplicating everything ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like that longer term, but afaik procmgr can't use the Go config runtime or the embedded schema. For config gates we only need env resolution for ~13 keys, so this PR keeps an explicit table that mirrors those schema env_vars (with schema comments cross-linking back). Happy to follow up with either schema-driven codegen for env_bindings.rs or a CI check that diffs ENV_BINDINGS against schema env_vars for gated keys. WDYT?

@pgimalac pgimalac Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema is a yaml schema with a well known format so I do think it could be used ?
Either read at build/run time, or to generate rust code for those configs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. That's what I meant by long term. I'll evaluate the effort to implement it now

Comment thread pkg/procmgr/rust/src/manager.rs
Comment thread pkg/config/schema/yaml/core_schema.yaml

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The reload path now leaves a failed config-managed process stopped even after a subsequent config reload supplies a valid command, removing the recovery behavior covered by the deleted test. The same diff also removes the schema defaults for sketch v3 routing and shadow settings while serializer code still reads them, which can produce an empty v3beta route or disable expected shadowing.

Open Bits AI session

🤖 Datadog Autotest · Commit 8cc535a · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cc535acbf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/config/schema/yaml/core_schema.yaml Outdated
Comment thread pkg/procmgr/rust/src/manager.rs Outdated
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 52.44% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 10e542e | Docs | Datadog PR Page | Give us feedback!

@jose-manuel-almaza

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2e1a30819

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pkg/procmgr/rust/src/config_gate.rs Outdated
Comment thread pkg/procmgr/rust/src/config_gate.rs
Comment thread pkg/procmgr/rust/src/config_gate/system_probe.rs Outdated
Comment thread pkg/config/setup/BUILD.bazel Outdated
jose-manuel-almaza added a commit that referenced this pull request Aug 12, 2026
Gate evaluation stays stubbed in the spawn-profiles PR; integration tests
that require real YAML gate behavior belong in jose/procmgr-config-gates.
@jose-manuel-almaza
jose-manuel-almaza force-pushed the jose/procmgr-config-gates branch from a06eeda to df5ab47 Compare August 14, 2026 14:16
Add an OperationGate so config reload and daemon shutdown block list,
describe, get_status, get_config, create, start, and stop with
FAILED_PRECONDITION while in progress.
Parse condition_config_any from processes.d and evaluate config gates
before auto-start; stub always-open gates are replaced in the next commit.
…R 2/4)

Restore core Agent SCM env cache, resolve_executable, tokio baseline env,
secret-backend win32 helpers, and Windows to_command_stdio helpers moved
out of the spawn-profiles PR so each stacked PR carries only what it uses.
Add Agent-parity config gate evaluation (YAML, env, fleet, system-probe
derivations) so processes.d definitions can gate on process_config and related
settings. Secret backend resolution lands in PR 3.
Revert accidental sketches-to-series rename in core_schema.yaml from the
config-gates split. Serializer shadow settings still read sketches.* keys.
Mirror condition_path_exists behavior so ${DD_CONF_DIR} and other
placeholders resolve before YAML gate evaluation.
serde_yaml leaves yes/on as strings while go.yaml.in/yaml/v2 treats them
as true. Apply YAML 1.1 parsing for file lookups only; env stays ParseBool.
Gate integration tests live in #54732 with the real evaluator after they
were removed from the spawn-profiles foundation PR.
Restore pkg/config/setup/BUILD.bazel comments from the base branch.
The #53249 split had added nonexistent schema.codegen flags; this PR
only needs the config_windows_test.go test target wiring.
…g gates

Use sp_bool_default for discovery.enabled so system_probe_config.enabled
derivation matches Go on Linux (true when unset) and ECS Fargate (false).
Resolve fleet policy directory from the config file being evaluated
(system-probe.yaml or datadog.yaml), matching Go system-probe loading.
…g gates

Load YAML with saphyr so YAML 1.1 bool spellings apply only to plain scalars;
quoted strings use ParseBool only, matching agent config semantics.
PATHEXT defaults use uppercase extensions while created files often use
lowercase; match existing files via read_dir when Path::is_file fails.
…ails

Warn and reuse the inherit/NUL stdio path when CreateFileW cannot open a
configured stdout/stderr file, matching Unix and tokio spawn behavior.
Tag queued retries with the originating process UUID and resolve the live
process by incarnation so a timer from a removed process cannot spawn a
replacement with the same name.
Move had_successful_run onto ManagedProcess so set_config can reset
RestartTracker without dropping post-success retry classification, and
add tests for reload plus failed respawn retry chains.
Drop module and API docs that restate obvious behavior; keep short notes
where Win32 spawn/auth semantics are non-obvious.
Treat config-changed reloads like a fresh PM start: reset counters on
set_config, start running/failed auto-start processes under boot gates,
and discard all pending crash timers from before the reload.
Align restart-policy log assertion with the current message format and
resolve rustfmt/clippy failures on the spawn-profiles branch.
Restore process.rs, spawn/mod.rs, and shutdown.rs from the reload-gate
branch so profile/user display and the unified stop path survive the
config-gates rebase onto main.
…ofiles

Keep config-gates on stop().await while adopting the ProcessManager reload
helper split from spawn-profiles, and drop the obsolete align-reload commit.
Replace outdated module comments that referenced hand-written Go
BindEnvAndSetDefault setup with schema YAML env_vars and schema.codegen.
Remove leftover merge markers in may_auto_start and align config-gate
integration tests with RuntimeHandles after the manager module split.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants